Outcome-driven Evaluation Metrics for Treatment Recommendation Systems

Treatment recommendation systems aim to providing clinical decision supports, e.g. with integration of Computerized Physician Order Entry (CPOE). One of the most significant issue is the quality of recommendations which needs to be quantified, before getting the acceptance from physicians. In computer science, such evaluations are typically performed by applying appropriate metrics that provides a comparison of different systems. However, a big challenge for evaluating treatment recommendation systems is that ground truth is only partially observed. In this paper, we propose an outcome-driven evaluation methodology, and present five metrics (i.e. precision, recall, accuracy, relative risk and odds ratio) with highlight of their statistic meanings in clinical context. The experimental results are based on the comparison of two well-developed treatment recommendation systems (one is knowledge-driven and based on clinical practice guidelines, while the other is data-driven and based on patient similarity analysis), using our proposed evaluation metrics. As a conclusion, physicians are less prone to comply with clinical guidelines, but once following guideline recommendations, it is much more likely to get good outcomes than not following.